custom headers

unknownmenz

I need to connect to one host proxy and it will assign me 1 random ip address from the pool, which is available for me (depends on country/size of package).

Let's say i have a package of 10 IP's.
if i assign to the host proxy, i will get one of the 10 IP's.
It won't change except on 17:00 evening and 05:00 morning. Unless i try to connect/request to the host proxy again.

daemons

@unknownmenz Just place anywhere before the Load action.

unknownmenz

@daemons i did that, setting the proxy is the first thing which is done before any other acitivty will be done.

daemons

@unknownmenz Good. Browser\Proxy, then Network\Set Header, then Browser\Load should all get the work done. Use Fiddler to double check.

unknownmenz

is there anything i need to put into "set header"? because i use it know so it forces X website to mobile device website

edit: so it visits a website like a android device (user agent)

daemons

@unknownmenz Just set X-ProxyMesh-IP equal to the IP from the preceding response header's X-ProxyMesh-IP. That should be it. According to your ProxyMesh docs. All X-headers are just a way to separate them from the standard. There's nothing to do with desktop or mobile. Servers only can determine that by the User-Agent string. I say it again, use Fiddler to double check everything, including the User-Agent headers. Then tweak if needed.

unknownmenz

@daemons Thanks a lot, i will try to get it solved. I will let you know.

unknownmenz

@daemons It seems not to work for me, it looks like everytime i use the load function another ip gets assigned from the pool.. with or without the headers i try to use.

And fiddler i try to work it out as i am new to a program like that.

daemons

@unknownmenz Probably it's time to contact their support. But anyway they'd need some logs. Fiddler is a way to go then. It's widespread, learn it.

BTW there are many other ways for servers to detect unwanted clients, not only headers, but first you need to get rid of your IP problem.

unknownmenz

@daemons said in custom headers:

@unknownmenz Probably it's time to contact their support. But anyway they'd need some logs. Fiddler is a way to go then. It's widespread, learn it.

BTW there are many other ways for servers to detect unwanted clients, not only headers, but first you need to get rid of your IP problem.

This is their response.

"
Another method for controlling the IP addresses is to use random string in the Proxy-Authorization header. The way it works is instead of passing in an encoded username:password in the authentication header, you pass in username:string:password, where string is a random string that you generate. As long as you keep using the same string, the proxy server will choose the same outgoing IP, as long as that IP is available. If the IP is no longer available, then a new IP will be chosen by the proxy server, and will be used for subsequent requests with the same string. Most client libraries support the Basic access authentication method, which means the only change required in your code, is to join the string to your username when providing the header value. However, for HTTPS requests, only some libraries support sending the Proxy-Authorization header with the initial CONNECT, such as Python requests. Other client libraries may require custom code to make the Proxy-Authorization header work for HTTPS requests.
"
But i do not understand it.. too technical for me.. could you explain it to me how to implement it? or explain in baby talk ?

(im so sorry for the bothering).

daemons

@unknownmenz Long story short, I've tested their approach on a dummy HTTPS site (Google) and found out that BAS's browser (it's a custom build of Chrome) does not support Proxy-Authorization headers in HTTPS CONNECT, but HTTP-client (it's curl-based) does.

You can find the two test cases attached 0_1518990005633_tests01.xml and 1_1518990005633_tests02.xml. The first one lacks any auth data in Fiddler logs at all, other one shows what we need (last row):

CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Connection: Keep-Alive
Proxy-Authorization: Basic dXNlcjoyczUwNjc1cTpwYXNz

So, in fact, it all depends on your goals and resources. If you can build your app around GET/POST requests then the above approach would work in BAS. But if the app depends on Javascript, user interaction, etc. then no.

But I think, the first approach with X-headers is not completely compromised yet. So did you study Fiddler logs?

unknownmenz

@daemons said in custom headers:

@unknownmenz Long story short, I've tested their approach on a dummy HTTPS site (Google) and found out that BAS's browser (it's a custom build of Chrome) does not support Proxy-Authorization headers in HTTPS CONNECT, but HTTP-client (it's curl-based) does.

You can find the two test cases attached 0_1518990005633_tests01.xml and 1_1518990005633_tests02.xml. The first one lacks any auth data in Fiddler logs at all, other one shows what we need (last row):
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Connection: Keep-Alive
Proxy-Authorization: Basic dXNlcjoyczUwNjc1cTpwYXNz
So, in fact, it all depends on your goals and resources. If you can build your app around GET/POST requests then the above approach would work in BAS. But if the app depends on Javascript, user interaction, etc. then no.

But I think, the first approach with X-headers is not completely compromised yet. So did you study Fiddler logs?

I just need to make sure i keep using the same Proxy IP while doing its process on a https website.

I wasn't be able to learn fiddler yet(even though the the program looks cool, the program makes me feel like a hacker LOL).

daemons

@unknownmenz That's what I'm talking about: the process. It all depends on it. If it allows sticking to GET/POST - then OK, go with BAS. If you need complicated user interaction stuff - then sorry no. But that's for the second approach. First you need to vet the first one. I'd be happy to help you more, but I think, it's not possible without further details on you task.

unknownmenz

@daemons said in custom headers:

@unknownmenz That's what I'm talking about: the process. It all depends on it. If it allows sticking to GET/POST - then OK, go with BAS. If you need complicated user interaction stuff - then sorry no. But that's for the second approach. First you need to vet the first one. I'd be happy to help you more, but I think, it's not possible without further details on you task.

Why wouldn't it be allowed to GET/POST?

The automated process makes use of "load/get element text/script logics/sleep/set and move label/move and click on element" and stuff like that.

I can send you the script in private if you want

daemons

@unknownmenz I might be wrong, but from what you say, it looks like this process does require a browser, i.e. simple GET/POST won't suffice. Yeah, you can send that to me in private.